Introduce the dataset manifest and remove layer information from the partition table #11423
+684
−90
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Web viewer failed to build.
Note: This comment is updated whenever you push a commit. |
abey79
changed the title
abey79
added
sdk-python
abey79
force-pushed
the
antoine/layer-table
branch
from
701aed1 to
1cd8a69
Compare
abey79
changed the title
There was a problem hiding this comment.
Pull Request Overview
This PR introduces the dataset manifest table functionality and removes layer-specific columns from the partition table. The dataset manifest provides layer-level metadata while the partition table now focuses solely on partition-level information.
- Adds new gRPC endpoints for dataset manifest schema and scanning operations
- Refactors partition table structure to remove layer information and add partition metadata
- Implements dataset manifest provider for DataFusion integration
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| rerun_py/src/catalog/dataset_entry.rs | Adds manifest() method to expose dataset manifest as DataFusion table |
| rerun_py/rerun_bindings/rerun_bindings.pyi | Python type hints for new manifest() method |
| crates/store/re_server/src/store.rs | Updates partition table schema removing layer columns and adding metadata |
| crates/store/re_server/src/rerun_cloud.rs | Implements placeholder gRPC handlers for dataset manifest endpoints |
| crates/store/re_redap_client/src/lib.rs | Adds error variant for dataset manifest schema operations |
| crates/store/re_redap_client/src/connection_client.rs | Implements client method for dataset manifest schema fetching |
| crates/store/re_protos/src/v1alpha1/rerun.cloud.v1alpha1.rs | Generated protobuf code for new dataset manifest endpoints |
| crates/store/re_protos/src/v1alpha1/rerun.cloud.v1alpha1.ext.rs | Schema definitions and helper methods for dataset manifest responses |
| crates/store/re_protos/proto/rerun/v1alpha1/cloud.proto | Protocol buffer definitions for dataset manifest endpoints |
| crates/store/re_datafusion/src/partition_table.rs | Adds TODO comment for deduplication |
| crates/store/re_datafusion/src/lib.rs | Exports new DatasetManifestProvider |
| crates/store/re_datafusion/src/dataset_manifest.rs | Implements DatasetManifestProvider for DataFusion integration |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
crates/store/re_protos/src/v1alpha1/rerun.cloud.v1alpha1.ext.rs
Outdated
Show resolved
Hide resolved
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related
What
Introduces gRPC endpoints and associated SDK method to access the dataset manifest table, which contains a row per layer. Also, remove most layer-related columns from the partition table.
This PR also attempts to solidify the notion that
Scan{PartitionTable|DatasetManifest}Responseis the One True Source(tm) of information on the returned dataframe's schema.The OSS server does not yet implement the dataset manifest (RR-2482).
extutilitiesDatasetManifest(fromLayerTable)update catalog?Add dataset manifest table to the catalog provider #11444